Apparatus, systems and methods for dynamic online kinematic adaptation of medical robots

ABSTRACT

In a robotic medical system comprising a staging kinematic chain coupled to a plurality of independently articulable robotic arms capable of motion with one or more degrees of freedom, a first configuration of robotic arms may be determined based on a first inverse kinematic model including the robotic arms and assuming a static staging kinematic chain. The first configuration may effectuate desired poses of instruments coupled to the robotic arms. A set of control parameter values associated with the robotic arms may be determined based on the first configuration, and, when a determined control parameter value falls outside a corresponding control parameter range, a staging kinematic chain pose and a second configuration of the robotic arms to effectuate the desired poses of the instruments may be determined. The second configuration is determined using a second inverse kinematic model that includes the robotic arms and assumes a mobile staging kinematic chain.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. ProvisionalPatent Application No. 62/924,905, entitled, “APPARATUS, SYSTEMS ANDMETHODS FOR DYNAMIC ONLINE KINEMATIC ADAPTATION OF MEDICAL ROBOTS,”filed Oct. 23, 2019, which is assigned to the assignee hereof, andincorporated by reference in its entirety, herein.

FIELD

The subject matter disclosed herein relates to robotic medical systems,devices, and methods for kinematically adapting medical robots tofacilitate medical procedures.

BACKGROUND

Robotic medical systems are used during minimally invasive ornon-invasive medical procedures such as imaging tissue, performingbiopsies, surgery, or other medical procedures. In some systems, arobotic arm, which may include multiple sections linked by joints, maybe used by an operator to manipulate a medical instrument coupled to therobotic arm during a medical procedure. Robotic arms may be attached toa common base, which may be immobile during performance of the medicalprocedure. In many instances, the medical procedures being performed canbe hampered because the motion of one or more sections of a robotic armmay be constrained due to physical limitations. For example, one or moreactuators may be at maximum extension and prevent arm motion in certaindirections. In general, such physical constraints may distract or limitmedical practitioners from the procedure being performed, lead to delaysand/or inefficiencies to adjust or reposition robots, and/or impactpatient safety because the medical practitioner may not have any apriori indication of motion limits of the robotic medical system until aspecific procedure has already commenced.

Accordingly, some embodiments disclosed herein enhance safety andimprove procedural efficiency, in part by facilitating instrument motionand control during medical procedures.

SUMMARY

In some embodiments, a method may be performed on a robotic medicalsystem comprising a staging kinematic chain capable of motion with oneor more degrees of freedom (DOF), wherein the staging kinematic chain iscoupled to a plurality of independently articulable robotic arms. Themethod may comprise: determining a first configuration of the roboticarms based on a first inverse kinematic model that includes the roboticarms and assumes that the staging kinematic chain is static, wherein thefirst configuration of the robotic arms effectuates one or morecorresponding desired poses of one or more instruments coupled to therobotic arms; determining a set of control parameter values associatedwith one or more of the robotic arms based on the first configuration;and determining, in response to at least one determined controlparameter value falling outside a corresponding control parameter range,a pose of the staging kinematic chain and a second configuration of therobotic arms to effectuate the one or more corresponding desired posesof the one or more instruments, wherein the second configuration isdetermined based on a second inverse kinematic model that includes therobotic arms and assumes that the staging kinematic chain is mobile.

In a further aspect, a robotic medical system may comprise: a stagingkinematic chain, capable of motion with one or more degrees of freedom(DOF), a plurality of independently articulable robotic arms coupled tothe staging kinematic chain, one or more instruments coupled to therobotic arms, and a processor operationally coupled to the stagingkinematic chain, the plurality of robotic arms, and the one or moreinstruments. The processor may be configured to: determine a firstconfiguration of the robotic arms based on a first inverse kinematicmodel that includes the robotic arms and assumes that the stagingkinematic chain is static, wherein the first configuration of therobotic arms effectuates one or more corresponding desired poses of theone or more instruments coupled to the robotic arms; determine a set ofcontrol parameter values associated with one or more of the robotic armsbased on the first configuration; and determine, in response to at leastone determined control parameter value falling outside a correspondingcontrol parameter range, a pose of the staging kinematic chain and asecond configuration of the robotic arms to effectuate the one or morecorresponding desired poses of the one or more instruments, wherein thesecond configuration is determined based on a second inverse kinematicmodel that includes the robotic arms and assumes that the stagingkinematic chain is mobile.

In another aspect, an apparatus may comprise staging kinematic chainmeans capable of motion with one or more degrees of freedom (DOF) and aplurality of independently articulable robotic arm means. The method maycomprise: means for determining a first configuration of the robotic armmeans based on a first inverse kinematic model that includes the roboticarm means and assumes that staging kinematic chain means is static,wherein the first configuration of the robotic arm means effectuatescorresponding desired poses of instrument means; means for determining aset of control parameter values associated with the robotic arm meansbased on the first configuration; and means for determining, in responseto at least one determined control parameter value falling outside acorresponding control parameter range, a pose of staging kinematic chainmeans and a second configuration of the robotic arm means to effectuatethe corresponding desired poses of the instrument means based on asecond inverse kinematic model that includes the robotic arms andassumes that the staging kinematic chain is mobile.

In some embodiments, a non-transitory computer-readable mediumcomprising instructions to configure a processor coupled to a roboticmedical system to: determine a first configuration of the robotic armsbased on a first inverse kinematic model that includes the robotic armsand assumes that the staging kinematic chain is static, wherein thefirst configuration of the robotic arms effectuates one or morecorresponding desired poses of the one or more instruments coupled tothe robotic arms; determine a set of control parameter values associatedwith one or more of the robotic arms based on the first configuration;and determine, in response to at least one determined control parametervalue falling outside a corresponding control parameter range, a pose ofthe staging kinematic chain and a second configuration of the roboticarms to effectuate the one or more corresponding desired poses of theone or more instruments, wherein the second configuration is determinedbased on a second inverse kinematic model that includes the robotic armsand assumes that the staging kinematic chain is mobile.

The methods disclosed may be performed by one or more processors, and/orrobotic medical devices, etc. The disclosed methods may be embodied oncomputer-readable media or computer-readable memory.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will be described, by way of example only,with reference to the drawings.

FIG. 1A shows an example diagram illustrating some components of anexample robotic medical system in accordance with certain embodimentsdisclosed herein.

FIG. 1B shows an example instrument coupled to a distal end of a roboticarm.

FIG. 1C shows an example endoscope, which may be coupled to a roboticarm, and used to introduce an instrument into a patient's body.

FIG. 1D shows a schematic block diagram illustrating some functionalblocks of an example robotic medical system in accordance with certainembodiments disclosed herein.

FIGS. 2A and 2B show views illustrating the deployment of robotic armson example robotic medical system during an example medical procedure.

FIG. 3A shows a flowchart of a method to facilitate medical robot motioncontrol during medical procedures in accordance with certain embodimentsdisclosed herein.

FIGS. 3B and 3C show platform 110 with staging kinematic chain S 118being moved from an initial position (in FIG. 3B) to a subsequentposition (in FIG. 3C) during a medical procedure in accordance withcertain embodiments disclosed herein.

FIG. 4 shows an exemplary computing subsystem to facilitate medicalrobot motion control during medical procedures.

Like reference numbers and symbols in the various figures indicate likeelements, in accordance with certain example embodiments. In addition,multiple instances of an element may be indicated by following a firstnumber for the element with a letter or with a hyphen and a secondnumber. For example, multiple instances of an element 112 may beindicated as 112-1, 112-2, 112-N etc. When referring to such an elementusing only the first number, any instance of the element is to beunderstood (e.g. element 112 in the previous example would refer toelements 112-1, 112-2, and/or 112-N).

DETAILED DESCRIPTION

Embodiments disclosed herein enhance safety and improve proceduralefficiency, in part by facilitating instrument motion and control duringmedical procedures, including robot assisted medical procedures such asendoscopy, laparoscopy, and/or endolumenal procedures.

Robotic medical systems are increasingly used during minimally invasiveor non-invasive medical procedures such as imaging tissue, performingbiopsies, surgery, diagnostics, etc. The term “non-invasive” relates tothe use of naturally occurring openings in the human body (e.g. mouth,nose, anus, urethral openings, etc.) to perform medical procedures. Theterm “minimally invasive” refers to the use of one or more smallincisions (relative to incisions conventionally used for a procedure) toperform a medical procedure. These medical procedures may includegastro-intestinal (GI) tract procedures, trans-oral procedures,colorectal procedures, urologic procedures, gynecological procedures,thoracic and/or pulmonary procedures, cardiac procedures, and variousother types of surgical and/or diagnostic procedures.

Robotic medical systems may include devices that use one or more arms(also referred to herein as “robotic arms”), where each arm may includemultiple sections linked by joints. On each arm, actuators coupled tojoints and arm sections, may be used to move and reorient arm sections(or sub-arms) to effect a desired pose of one or more medicalinstruments coupled to a distal end of the arm. The instruments may beused to facilitate and/or perform medical procedures. A proximal end ofeach arm on the robotic medical system may be coupled to a base commonto the arms. Conventionally, the base is immobilized during performanceof the medical procedure. Further, the common base may be coupled to aplatform, which may house electronics and other controls to power andoperate the one or more arms and the base.

In many instances, medical procedures being performed can be hamperedbecause the motion of one or more sections of a robotic arm may beconstrained due to physical limitations. For example, one or moreactuators on an arm may be at maximum extension and prevent arm motionin certain directions. In general, such physical constraints may: (a)distract or limit medical practitioners from the procedure beingperformed, (b) lead to delays and/or inefficiencies to adjust orreposition the platform and/or robotic arms, and/or (c) impact patientsafety because the medical practitioner may not have any a prioriindication of motion limits of the robotic medical system until aspecific procedure has already commenced. For example, in a roboticmedical system with multiple arms, repositioning a platform during amedical procedure may involve, among other actions, removal of allinstruments from a patient's body, repositioning of the platform,repeating any set up procedures, re-navigating instruments to the priorlocations within the patient's body—all prior to recommencing. Thus, thetime taken for the medical procedure may be detrimentally impactedand/or operator cognitive load may increase significantly depending onthe complexity of the above tasks. In addition, such distractions canlead to errors, affect patient safety or recovery time, increaseprocedure cost, and/or decrease medical practitioner availability forother patients/procedures. In sum, instrument motion and positioningconstraints, which may occur because of actuator and/or jointlimitations on an arm, can detrimentally affect the duration,performance, efficiency, cost, and safety of medical proceduresperformed using a robotic medical system.

In some embodiments, a robotic medical system may facilitateeffectuation of instrument poses in response to an existing or predictedsystem state by facilitating motion of a staging kinematic chain induring a medical procedure in addition to movement of robotic arms. Akinematic chain may be viewed as an assembly of one or more rigid parts.The component parts may be connected by articulated joints. The pose ofa kinematic chain may be determined from (or represented by) the pose(s)of each of its component parts. For example, the poses of individualparts of a kinematic chain along with joint articulation parameters maybe used to determine the pose of a kinematic chain.

In some embodiments, a robotic medical system may comprise a stagingkinematic chain (S) capable of motion with one or more degrees offreedom (DOF), wherein the staging kinematic chain (S) may be coupled toa proximal end of a plurality of independently articulable robotic armsA={A_(r)|1≤r≤N, N≥2}, where A_(r) is a robotic arm and N represents thenumber of robotic arms. Each robotic arm A_(r) may be coupled, at itsdistal end, to one or more instruments (also referred to herein as “endeffectors”) E_(k_r), 1≤k≤M_(r), where M_(r), is the number ofinstruments coupled to the distal end of robotic arm A_(r). Further,each instrument E_(k_r) may be associated with a corresponding poseP_(k_r). The pose P_(k_r) of instrument E_(k_r) refers to the positionand orientation of instrument E_(r) in task space.

Thus, for a robotic arm A_(r), the set of instruments may be given byE_(r)={E_(k_r)|1≤k≤M_(r)} and the corresponding poses byP_(r)={P_(k_r)|1≤k≤M_(r)}. Further, the set of instruments may bewritten as E={E_(r)|1≤r≤N, N≥2} with corresponding instrument poseswritten as P={P_(r)|1≤r≤N, N≥2}.

In some embodiments (e.g. where k=M_(r)=1 for all r), each robotic armA_(r) may be coupled to a single distinct corresponding instrument,which may be written as E_(k_r)=E_(1_r)=E_(r) and E={E_(r)|1≤r≤N, N≥2}.Similarly, the pose associated with the corresponding instrument may bewritten as P_(k_r)=P_(1_r)=P_(r) and P={P_(r)|1≤r≤N, N≥2}. Forillustrative purposes only, in the examples below, each robot arm A_(r)is assumed to be coupled to a single distinct instrument E_(r) with poseP_(r) so that E={E_(r)|1≤r≤N, N≥2} and P={P_(r)|1≤r≤N, N≥2}. However, itis understood that the apparatus and techniques disclosed may be appliedwhen robot arms are coupled to more than one instrument.

In some embodiments, at a first time, a first correspondingconfiguration C₁={C_(1_r)|1≤r≤N} of the robotic arms A may be determinedto effectuate corresponding desired poses P of instruments E coupled tothe robotic arms A, wherein the first corresponding configuration C₁ ofthe independent arms may be determined based on a first inversekinematic model that includes the robotic arms and assumes that thestaging kinematic chain (S) is static. For a given robotic arm A_(r),the first corresponding robot arm configuration may written asC_(1_r)={c_(1r_f)|1≤f≤F_(r)}, where c_(1r_f) represents the firstconfiguration of joint f associated with robotic arm A_(r) and F_(r)represents the number of joints on robotic arm A_(r). Each robotic armA_(r) may comprise one or more joints so that 1≤f≤F_(r). Thus, the poseP_(r) of instrument E_(r) coupled to the robotic arm A_(r) may bedetermined by the configuration C_(1_r) of joints associated with therobotic arm A_(r). The configuration C_(1_r) of a robotic arm (alsoreferred to herein as joint configuration) refers to the position valueof each joint associated with the robotic arm A_(r) in joint space.Joint space is defined by a vector whose components are thetranslational and/or angular displacements of each joint of a roboticlink relative to a (joint space) frame of reference. The configurationof a joint on the robotic arm A_(r) may be determined by actuator statesassociated with each joint. Thus, when the staging kinematic chain isstatic, the configuration C_(1_r) of an arm A_(r) may be determined bythe configuration of joints on the arm. Accordingly, the jointconfiguration C_(1_r) (in joint space) may be represented as a vector ofthe actuator values (e.g. with one actuator value per actuator), Theactuator values may include angular values.

Kinematic models use mathematical relationships between position,velocity, and acceleration for joints and/or robot arm sections todetermine unknowns. Based on a kinematic model for the robotic medicalsystem, actuator inputs may be determined to achieve actuator statesthat move, rotate, and/or place joints in the robotic arm and/orinstruments (or end effectors) coupled to the robotic arm in appropriate(desired) poses. Forward kinematics determines the poses ofinstruments/end effectors coupled to a robot arm based on known jointconfigurations associated with the robotic arm. Inverse kinematicsdetermines a configuration of a robot arm (or joint configurations forthe robot arm) based on a known (or desired) poses of instruments/endeffectors coupled to the robot arm.

A set of control parameter values V={V_(r)|1≤r≤N}, associated with therobotic arms A may be determined based on the first correspondingconfiguration C₁, where, for each robotic arm A_(r), the set of controlparameter values V_(r) may be written as V_(r)={ν_(rq)|1≤q≤Q_(r)}, whereQ_(r) is the number of control parameter values associated with robotarm A_(r). In some embodiments, in response to the value of at least onedetermined control parameter ν_(rq) falling outside a correspondingcontrol parameter range (e.g. ν_(rq)<T1 or ν_(rq)>T2), a configurationCs (which may correspond to a pose P_(S)) of the staging kinematic chainand a second corresponding configuration of the robotic armsC₂={C_(2_r)|1≤r≤N} to effectuate the corresponding desired poses P ofinstruments E may be determined, wherein the second correspondingconfiguration C₂ of the robotic arms A is determined based on a secondinverse kinematic model that includes the robotic arms A and assumes amobile staging kinematic chain S.

In some embodiments, the first inverse kinematic model may be used todetermine the first corresponding configuration C₁ of the robotic arms Aby determining, for each robotic arm A_(r), corresponding actuatorpositions. For example, the configuration of arm A_(r) may be determinedby determining actuator positions (e.g. at various joints) associatedwith robotic arm A_(r). In some embodiments, the corresponding actuatorpositions for each robotic arm A_(r) may be determined independently ofother robotic arms A_(u), u≠r and independently of the staging kinematicchain (S). In some embodiments, the first inverse kinematic model may bebased on N independent kinematic chains, wherein each kinematic chaincorresponds to a robotic arm A_(r) in the plurality of robotic arms A.

In some embodiments, at the first time, the second inverse kinematicmodel may be used to determine a second corresponding configuration C₂of the robotic arms A based on: (i) the degrees of freedom (DoFs)available to the staging kinematic chain (S) and (ii) available DOFs toeach independent arm A_(r). In some embodiments, the second inversekinematic model may be based on a single kinematic chain comprising thestaging kinematic chain (S) and the plurality of robotic arms A.

In some embodiments, the switch from the first correspondingconfiguration of the robotic arms C₁ to the second correspondingconfiguration of the robotic arms C₂ at the first time, may beeffectuated dynamically during a medical procedure. In some embodiments,the robotic medical system may dynamically switch between the firstinverse kinematic model and the second inverse kinematic model. Forexample, the robotic medical system may dynamically switch to the firstinverse kinematic model upon effectuation of the corresponding desiredposes P_(k_r) of instruments E_(k_r) coupled to the robotic arms A basedon the second inverse kinematic model. For example, once thecorresponding desired poses of instruments P_(k_r) of instrumentsE_(k_r) coupled to the robotic arms A have been effectuated (based onthe second inverse kinematic model), the robotic medical system maydynamically switch to the first inverse kinematic model.

In some embodiments, the dynamic switch between the first inversekinematic model to the second inverse kinematic model (or vice versa)may occur within a control tick. In some embodiments, correspondingdesired poses P_(k_r) of instruments E_(k_r) coupled to the robotic armsA (e.g. as determined by the first inverse kinematic model or the secondinverse kinematic model) may be effectuated within a control tick. Themethods described herein may be performed automatically (e.g. byprocessor(s) associated with the robotic medical system) based on thecorresponding desired poses of instruments coupled to the robotic armsand without further user-input.

FIG. 1A shows an example diagram illustrating some components of anexample robotic medical system 100 in accordance with certainembodiments disclosed herein. Robotic medical system 100 is merely anillustrative example and the techniques described herein may be appliedto various other types of systems. FIGS. 1A-1D are merely illustrativenon-limiting examples of systems and mechanisms (e.g. visual interfaces,control interfaces, instrument controllers/manipulators, adaptors, etc.)to facilitate description of example robotic medical system 100.Although some examples are described herein, these examples are notintended to be limiting.

In some embodiments, robotic medical system 100 may comprise a cart orrobotic medical device platform 110 (hereinafter “platform 110”), whichmay be mobile when offline and positioned in proximity to a patient.While platform 110 can be moved (e.g. when offline), typically, platform110 is stationary and is not moved during a medical procedure (e.g. whenonline). In some embodiments, platform 110 may comprise stagingkinematic chain S 118, which may have one or more degrees of freedom.For example, as shown in FIG. 1A, staging kinematic chain S 118 may becapable of movement vertically along axis 120. However, in someembodiments, staging kinematic chain S 118 may have additional degreesof freedom and may be able to move and/or rotate in other directions.Typically, in conventional systems, a staging arm or staging kinematicchain is not moved (when online) during the medical procedure and isonly used (offline) during set up of the system. Once set up, inconventional systems, the staging kinematic chain remains immobile forthe duration of the medical procedure.

The term “degrees of freedom” refers to the number of independentparameters that determine the pose of an object. The term “pose” refersto the position (e.g. X, Y, Z coordinates) and orientation (e.g. roll ϕ,pitch θ, and yaw ψ) of an object relative to a frame of reference. Pose(in Cartesian or “task space”) may be specified as a 6Degrees-of-Freedom (DOF) pose, which may include positional coordinates(e.g. X, Y, Z) and orientation information (e.g. roll, pitch, and yaw)relative to a frame of reference (e.g. for the task space). Thepositional coordinates (X, Y, Z) are also referred to herein asCartesian coordinates. Orientation can be specified in terms ofdirectional cosines. Various other coordinate systems (e.g. spherical,etc.) may also be used to describe pose. The term “instrument pose” or“end effector pose” may refer to the position and orientation of aninstrument and/or end effector relative to a frame of reference. In someinstances, instruments may be coupled to end effectors 116. Thus, achange in end effector pose may be reflected in corresponding changes toinstrument pose. In some embodiments, the frame of reference may becentered on platform 110. The term “camera pose” or “image sensor pose”may refer to the position and orientation of the image sensor relativeto a frame of reference. In some embodiments, the frame of reference maybe image sensor centric and may be used to determine the position of oneor more instruments (not shown in FIG. 1A) and/or other componentsrelative to the (image sensor centric) frame of reference.

As shown in FIG. 1A, a distal section of staging kinematic chain S 118is coupled to the corresponding proximal end 114-r of each of theplurality of independently articulable robotic arms A={A_(r)|2≤r≤N,N≥2}, where A_(r) 112-r represents the r^(th) robotic arm and Nrepresents the number of robotic arms. A proximal section of the stagingkinematic chain S 118 may be coupled to the body of platform 110 in amanner that facilitates motion of staging kinematic chain S 118 with oneor more degrees of freedom.

The term “independently articulable” in relation to a robotic arm A_(r)112-r indicates that the robotic arm A_(r) 112-r may be moved (i)independently of any other robotic arm A_(u) 112-u, u≠r, where A_(r),A_(u)εA so that movement of robotic arm A_(r) 112-r may occur without achange to the configuration and/or position of any other robotic armA_(u) 112-u; and (ii) independently of staging kinematic chain S 118.Further, as shown in FIG. 1A, a robotic arm A_(r) 112-r may comprise armsections between joints 113. Joints 113 (e.g. 113-1 . . . 113-N) mayinclude actuators that may move and orient the arm sections to realize acorresponding desired pose P_(r) for instrument E_(r) coupled to therobotic arm A_(r). Each robotic arm A_(r) 112-r may be coupled to asingle distinct corresponding instrument E_(r) and each instrument E_(r)may be associated with a corresponding desired pose P_(r). InstrumentE_(r) (not shown in FIG. 1A) may be coupled to a distal end of 116-r(116-1 . . . 116-N) of a corresponding robotic arm A_(r) 112-r. In someembodiments, distal end of 116-r of robotic arm A_(r) 112-r may includean instrument control/manipulator, which may be used to manipulate andcontrol instruments E_(r) coupled to distal end 116-r of robotic armA_(r) 112-r. For example, an endoscope with instrument E_(r) may becoupled to distal end 116-r of robotic arm A_(r) 112-r using theintegrated instrument control/manipulator and a desired pose P_(r) of aninstrument E_(r) may be attained using electro-mechanical articulation(e.g. through commands/control input from a physician operator).

In general, a configuration of actuators and/or joints to effect adesired pose P_(r) for an instrument E_(r) coupled to arm A_(r) 112-rmay be denoted as C_(w)={C_(w_r)|1≤r≤N}, where C_(w_r) (in joint space)represents a configuration of robot arm A_(r) 112-r. When the stagingkinematic chain S 118 is static, a configuration of actuators and/or armsections to effect a desired pose P_(r) for an instrument E_(r) coupledto arm A_(r) 112-r may be given by C_(w_r). Because the same desiredpose P_(k_r) for an instrument E_(k_r) coupled to an arm A_(r) may beeffected with different arm configurations, each distinct configurationof the robotic arms that effects the same desired instrument poseP_(k_r) is denoted by the subscript “w” in C_(w). When the stagingkinematic chain is moved, the pose P_(r) for an instrument E_(r) coupledto arm A_(r) 112-r may be determined by both C_(w_r) and C_(S), whereC_(S) represents a configuration of the staging kinematic chain S 118(which may correspond to a pose Ps of the staging kinematic chain S118).

Example platform 110 may include electronic, electro-magnetic, and/orelectro-mechanical systems to power, control, and operate stagingkinematic chain S 118 and the plurality of robot arms A 112. In someembodiments, robotic medical system 100 may also include console 125,which may be coupled to platform 110. For example, console 125 may beelectrically and communicatively coupled (e.g. wired or wirelessly) toplatform 110.

Example console 125 may include visual interface 135, which may displayimages including stereoscopic images from image sensors/cameras coupledto one or more robotic arms A_(r) 112-r. Visual interface 135 is merelyan illustrative non-limiting example and various other visual interfacesmay be used. Example visual interface 135 may also display configurationinformation for robotic medical system 100 and other relevantinformation (including graphical information) related to the medicalprocedure being performed. In some embodiments, visual interface 135 maytake the form of a headset, which may be wirelessly coupled to console125 and be capable of displaying stereoscopic images and otherinformation. Visual interface 135 may provide a real time indication ofinstrument pose (position and orientation). In some embodiments,currently selected components (e.g. robot arms 112) and/or the currentfunction selected or being performed using user interface 140 may bedisplayed or indicated to the user (e.g. as an overlay) in visualinterface 135.

In some embodiments, visual interface 135 may include a touchscreen thatmay accept user input. For example, visual interface 135 may facilitateviewing and interaction with a prior computer tomography (CT) scan andallow a physician/operator to plan a pathway through a patient's body.In some embodiments, visual interface 135 may also facilitate viewing ofprior CT scan(s), segmentation of pathways to facilitate visualizationand path planning, target selection and identification. In someembodiments, path planning may be based on prior CT scan and/or may bemanually modified or generated. In some embodiments, visual interface135 may also provide set up and/or other instructions (including voiceinstructions), facilitate menu selection and input, system configurationinformation, etc.

In some embodiments, robotic medical system 100 may include example userinterface 140, which may be used by an operator to select, activate, andcontrol instruments E_(r) coupled to robotic arms A_(r). 112-r. Userinterface 140 is merely an illustrative non-limiting example and variousother user interfaces may be used. Further, in some embodiments,instruments E_(r), coupled to robotic arms A_(r). 112-r may also beautonomously driven (e.g. by a processor or control system based oncomputer program code). User interface 140 may be communicativelycoupled to platform 110 and/or console 125. In some embodiments, userinterface 140 may include haptic feedback to alert an operator inrelation to various conditions related robotic medical system 100. Forexample, haptic feedback (e.g. based on the position of instruments 105relative to the current FOV of image sensors 110) may be received by theuser via a haptic interface on user interface 140.

User interface 140 may also include controls such as joysticks, knobs,buttons, etc. which may be used to select a robotic arm A_(r) 112-rand/or an instrument E_(r), or to control the pose P_(r) of the selectedrobotic arm A_(r) 112-r and/or instrument E_(r). Operator selections andmovement of instruments may be displayed using visual interface 135. Insome embodiments, a real time indication of instrument pose (positionand orientation) may be overlaid over images captured by camera(s)coupled to the instrument E_(k_r). User interface 140 may alsofacilitate manipulation of and utilization of functionality associatedwith a selected instrument E_(r), such as irrigation, aspiration,performing biopsies, taking snapshots, activating pre-programmedsequences, etc. For example, pre-programmed sequences may be based on apre-operative computed tomography (CT) scan, which may be integratedinto an intra-operative interface. In some embodiments, instrument posemay be determined based, on sensory information (e.g. electromagneticand other sensors) and on information from a prior CT scan.

FIG. 1B shows an example instrument 105-1 coupled to a distal end 116-1of a robotic arm 112-1 (not shown in FIG. 1B). As outlined above, theexample mechanisms (e.g. interfaces, instrumentcontrollers/manipulators, adaptors, catheters, instruments, etc.)described herein are merely illustrative and no limitation is intended.The mechanisms used may depend on the type procedure being performed,and/or other parameters. Thus, the use of other mechanisms (e.g. withvarious properties—rigid, articulable, flexible, etc., and/or forms,and/or capabilities) that may differ in one or more aspects from theillustrative examples is envisaged in conjunction with techniquesdescribed herein,

As shown in FIG. 1B distal end 116-1 of a robotic arm 112-1 (not shownin FIG. 1B, #may include an instrument control/manipulator, which may beused to manipulate and control instruments (such as instrument 105-1)coupled to distal end 116-1 using endoscope 108 (shown with dashed linesin FIG. 1B). In FIG. 1B, instrument E_(1_1) 105-1 is shown as a needle105-1. In general, various other instruments may be coupled to distalend 116-1 and/or instrument control/manipulator of robotic arm 112-1.Endoscope 108 may include a hollow main sheath through which one or moreinstruments may be inserted into a patient's body. Endoscope 108 mayinclude an image sensor (e.g. a stereo camera) and other sensors, whichmay be used to navigate sections of endoscope 108 and instrument(needle) E_(1_1) 105-1 through a patient's body. For example, imagescaptured by the image sensor may be displayed on visual interface 135and used during navigation. As shown in FIG. 1B, in some embodiments,robotic medical system may include an adapter 124 and/or othercomponents that may facilitate introduction of endoscope 108 and/orinstruments (needle) E_(1_1) 105-1 into the patient's body. The adaptorsand/or other components may depend on the medical procedures beingperformed and/or on the instruments being used. To place instruments(e.g. needle) E₁ 105-1 into a desired pose P₁, actuators (not shown inFIG. 1B) in robotic arm A₁ 112-1 and/or staging kinematic chain S 118may be configured in a configuration C_(w_1) and C_(S), respectively, tomove and orient arm sections of robot arm A₁ 112-1 to effect the desiredpose P₁.

In some embodiments, robotic medical system 100 may also includeelectro-magnetic field generator/sensor 122, which may generate anelectromagnetic field and sense changes to the electromagnetic field tohelp determine the position of instruments 105-1. In some embodiments,additional sensors coupled to the endoscope 108 and/or instruments 108may be used, at least in part, to track endoscope 108 and/or determine apose of instruments 105.

FIG. 1C shows a section of an example endoscope 108, which in someinstances, may comprise a flexible hollow main sheath that includes aworking channel, which may be used to introduce instruments such asinstrument 105-2 (e.g. a cauterizing knife) into a patient's body. Insome embodiments, instruments 105 may be coupled to a flexiblearticulable instrument arm 107 such as instrument arm 107-2, which maybe moved and/or oriented based on user input. Instrument arms 107 may beretracted in to the working channel when not in use, or duringnavigation of the endoscope, and may be activated and extended inaccordance during use in accordance with user input. In someembodiments, the working channel (e.g. housing instrument 105 and/orseparate working channels within the main sheath of endoscope 108) mayalso include separate tubing lines (not shown in FIG. 1C) forirrigation, aspiration, etc. FIG. 1C is merely a non-limitationillustrative example. For example, instruments 105 may also be coupledother types of arms (e.g. rigid) depending on the procedure beingperformed.

In some embodiments, endoscope may also include image sensors 104 (whichmay be stereoscopic) and light sources 106 for illumination of thesurrounding environment. Image sensors 104 may include cameras, CCDsensors, or CMOS sensors, which may transform an optical image into anelectronic or digital image and may send captured images to a processor.In some embodiments, image sensors 104 may capture color images, whichprovide “color information,” while “depth information” may be providedby a depth sensor. The term “color information” as used herein refers tocolor and/or grayscale information. In general, as used herein, a colorimage or color information may be viewed as comprising 1 to G channels,where G is some integer dependent on the color space being used to storethe image. For example, an RGB image may be viewed as comprising threechannels, with one channel each for Red, Blue, and Green information. Insome embodiments, depth information may be captured using depth sensors(active or passive). The term “depth sensor” is used to refer tofunctional units that may be used to obtain depth informationindependently and/or in conjunction with image sensors 110.

In some embodiments, image sensors 104 may form a (passive) stereoscopicimage sensor, and may capture stereoscopic images, which may be used todetermine depth information. Accordingly, in some embodiments, capturedimages may include stereoscopic or 3D images with depth information. Forexample, pixel coordinates of points common to both image sensors (e.g.104-1 and 104-2) in a captured image may be used along withtriangulation techniques to obtain per-pixel depth information. In someembodiments, the depth information may be used, at least in part, todetermine the pose of instruments 108 (e.g. relative to image sensors104 and/or another frame of reference). In some embodiments, visual posedetermination techniques may be used in conjunction with informationobtained from electromagnetic sensors (e.g. coupled to instruments 105)to determine a current pose of instrument 105. Although, image sensors104 and light source 106 are shown mounted to the body of endoscope 108,various other configurations are possible. For example, in someembodiments, cameras 104 and light source 106 may also be mounted onindividual flexible articulable extensible arms.

FIG. 1D shows a schematic block diagram illustrating some functionalblocks of an example robotic medical system 100 in accordance withcertain embodiments disclosed herein. The functional blocks shown inFIG. 1D and their distribution between platform 110, console 125, anduser interface 140 are merely examples and various other configurationsof the functional blocks may be used.

As shown in FIG. 1D, robotic medical system 100 may comprise console125, which may be coupled to visual interface 135 (e.g. a 3D display orstereoscopic display); user interface 140, which may be coupled tohaptic interface 142, and user input/system controls 146 (hereinafterreferred to as “system controls 146”). The above blocks (console 125,visual interface 135; user interface 140, haptic interface 142, andsystem controls 146) have been described above in relation to FIGS.1A-C).

Robotic medical system 100 may further include processor(s) 150, memory170, and robot control system 160. FIG. 1D is merely exemplary and thefunctionality associated with blocks shown in FIG. 1B may be combined(e.g. into a single block), or the functionality in a block may bedistributed across several blocks. For example, the functionalityassociated with robot control system block 160 may be integrated withinprocessor(s) 150 block or vice versa. As another example, thefunctionality associated with processor(s) 150 block may be combinedwith user interface block 130 or vice versa. As a another example, thefunctionality associated with processor(s) 150 block, user interfaceblock 130, and robot control system block 160 may be combined. As afurther example, the functionality associated with processor(s) 150block may be distributed between robotic medical device control systemblock 160 and user interface block 130. For example, each block may havelocal processor(s) 150 that cooperate (e.g. via commands, signals, andexchange of messages) to enable the functionality described herein.Further, functionality associated with one or more of the functionalblocks described may be combined (e.g. in platform 110, console 125, oruser interface 140) or distributed (e.g. between platform 110, console125, and/or user interface 140) as appropriate.

Haptic Interface 142 may provide haptic feedback to signal conditionsassociated with instruments 105 and/or robotic arms 112, which may bebased on user motion input 143. For example, haptic feedback 141 may bebased on the position of instruments 105 relative to tissue boundariesand/or the current FOV of image sensors 104 and/or a direction of motionof the instruments 105. Haptic feedback 141 may be received by the uservia haptic interface 142. User motion (e.g. via a joystick on userinterface 140) may be received by processor(s) 150 as user motion input143 and communicated to robotic control system 160, which may processthe received input and provide appropriate signals to articulate/movethe selected and/or active component (one or more of staging kinematicchain S 118, robot arms 112, image sensors 104, and/or instruments 105).

In some embodiments, a user may control and move end effectors and/orinstruments 105, while robotic arms 112 and/or staging kinematic chain118 may be moved automatically (e.g. based on input from processor (s)150 and/or robot control system 160) to attain the user desired endeffector/instrument pose. User Input/System controls block 146 may alsoprovide robot control input 147 to processor(s) 150, which may beprocessed and communicated to robot control system 160. Robot controlsystem 160 may process the received robot control input 147 and provideappropriate signals to select, activate, deploy, retract, disable, moveand/or orient one or more of: (i) robotic arms 112, and/or (ii)actuators coupled to robotic arms 112 and/or staging kinematic chain S108, and/or arm joints and/or instruments 105, and/or (iii) instruments105; and/or (iv) other components (e.g. one or more of image sensors110, light sources, etc.).

In some embodiments, images captured by image sensors 104 (e.g. coupledto corresponding instruments 105) may be processed and displayed onvisual interface 135. User motion input 143 (e.g. to move instruments105) and robot control input 147 (e.g. to control operation ofinstruments 105) may occur in response to the images displayed on visualinterface 135.

In some embodiments, processor(s) 150 may be coupled to memory 170. Asshown in FIG. 1B, memory 170 may include motion models block 175, whichmay be used by processor(s) 150 to control actuators coupled to one ormore of: robotic arms 112 and/or staging kinematic chain S 108, and/orarm joints and/or instruments 105 to effect a desired pose. Actuatorconfiguration may facilitate control of: motion and pose of the stagingkinematic chain S 108, the motion and pose of robot arms 112-r, and themotion and/or pose of instruments 105. Motion models block 175 mayinclude forward kinematic models, inverse kinematic models and/orcalibrated instrument control models.

In some embodiments, motion models block 175 may include both forwardand inverse kinematic models pertaining to robotic medical system 100.In some embodiments, motion models block 175 may include a first inversekinematic model. The first inverse kinematic model may include therobotic arms A 112 and may assume that the staging kinematic chain S 118is static when determining at a time t1, a first correspondingconfiguration C₁={C_(1_r)|1≤r≤N} of the robotic arms A 112 thateffectuates corresponding desired poses P of instruments E coupled tothe robotic arms A 112.

In some embodiments, the first inverse kinematic model may be used todetermine a first corresponding configuration C₁ of the robotic arms A112 by determining, for each robotic arm A_(r) 112-r, correspondingactuator positions. For example, the configuration of arm A_(r) 112-rmay be determined by determining actuator states at various jointsassociated with robotic arm A_(r) 112-r. In some embodiments, thecorresponding joint positions (or actuator states)C_(1_r)={c_(1r_f)|1≤f≤F_(r)} for each robotic arm A_(r) in joint spacemay be determined independently of other robotic arms A_(u), u≠r andindependently of the staging kinematic chain (S). In some embodiments,the first inverse kinematic model may be based on N independentkinematic chains, wherein each kinematic chain corresponds to a roboticarm A_(r) in the plurality of robotic arms A.

In some embodiments, motion models block 175 may further include asecond inverse kinematic model. The second inverse kinematic model mayinclude the robotic arms 112 and may assume mobility of the stagingkinematic chain (S), when determining at a time t1, a secondcorresponding configuration a second corresponding configuration of therobotic arms C₂={C_(2_r)|1≤r≤N} and a configuration Cs to effectuate thecorresponding desired poses P_(k_Ar) of instruments E_(k_r) coupled tothe robotic arms, where Cs is the configuration staging kinematic chainS 118 (which may correspond to a pose Ps of the staging kinematic chainS).

In some embodiments, the second inverse kinematic model may be used todetermine the second corresponding configuration C₂ of the robotic armsA 112 based on a single kinematic chain, wherein the single kinematicmodel combines available degrees of freedom (DoFs) corresponding to allthe sub-arms and available DoFs of the staging kinematic chain into thesingle kinematic chain.

Memory 170 may also store and determine configuration information 177.Configuration information 177 may include current poses P of instruments(or end effectors) 105, current and/or desired configuration/stateinformation (e.g. for actuators, robot arms 112, staging kinematic chainS 118, instruments 105, etc.), information from sensors coupled torobotic medical system 100 (e.g. sensors coupled to staging kinematicchain S 118, robotic arms 112, instruments 105, instrument arms 106,and/or actuators, etc.), and control parameter values for actuators,robot arms 112, staging kinematic chain S 118, etc. The term controlparameters may include one or more of: metrics related to the operationof one or more components of robotic medical system 100; and/or systemlevel parameters for robotic medical system 100; and/or parameter rangesfor one or more of: actuators, robot arms, and/or instrument motion,and/or disallowed states (e.g. to avoid robot arm collisions);parameters related to the medical procedure being performed (e.g.whether staging kinematic chain may be moved during the procedure and/orportions of the procedure) etc. The configuration information may beused by motion logic block 169 to control actuators and effectuatedesired instrument poses P (e.g. based on user input 146).

Motion logic block 179 may obtain and/or determine a set of controlparameter values V={V_(r)|1≤r≤N}, associated with each robotic arm A_(r)112-r based on a current configuration and/or a desired configuration ofthe robotic arm 112-r (e.g. in relation to desired instrument poses asdetermined by motion model 165). For example, the control parametervalues may be obtained based on the first corresponding configurationC₁, where, for each robotic arm A_(r) 112-r, the corresponding set ofcontrol parameter values may be written as V_(r)={ν_(rq)|1≤q≤Q_(r)}. Insome embodiments, in response to the value of at least one determinedcontrol parameter ν_(rq) falling outside a corresponding controlparameter range (e.g. ν_(rq)<T1 or ν_(rq)>T2, where T1≤T2), motion logicmay invoke second inverse kinematic model, which may result inmotion/change of pose of staging kinematic chain S 118. Accordingly,motion logic block 169 may determine a pose of the staging kinematicchain (P_(S)) and a second corresponding configuration of the roboticarms C₂={C_(r_2)|1≤r≤N} to effectuate the corresponding desired posesP_(k_Ar), as outlined above.

In some embodiments, at a first time t1, the switch from the firstcorresponding configuration of the robotic arms C₁ to the secondcorresponding configuration of the robotic arms C₂ may be effectuateddynamically during the medical procedure. In some embodiments, aftereffectuation (e.g. at time t1) of the desired poses based on the secondinverse kinematic model, motion logic 169 may initiate a switch back tothe first inverse kinematic model. In some embodiments, the firstinverse kinematic model may be a default option, and the second inversekinematic model may be invoked whenever the value of at least onedetermined control parameter ν_(rq) falls outside a correspondingcontrol parameter range.

In some embodiments, the dynamic switch between the first inversekinematic model to the second inverse kinematic model (or vice versa)may occur within a control tick. In some embodiments, configurations C1(of robotic arms A 112) or C2 (of robotic arms A 112 and stagingkinematic chain S 118), which correspond to desired poses P ofinstruments E coupled to the robotic arms A 112 may be effectuatedwithin a control tick. The methods described herein may be performedautomatically (e.g. by processor(s) 150 and/or robot control system 160associated with the robotic medical system) based on the correspondingdesired poses of instruments coupled to the robotic arms and withoutfurther user-input.

The term “control tick” refers to some specified duration of time (orprocessor cycles) within which motion related computations are completedand new setpoints are transmitted to the actuators. Completing actionswithin the “control tick” may ensure that the system is perceived asbeing responsive (with a lower latency) to operator input. In someembodiments, performing the switching between the first inversekinematic model and the second inverse kinematic model (or vice versa)within a control tick may ensure a seamless user experience.

In some embodiments, motion models block 175 may include calibratedinstrument control models, which may be used to estimate an instrumentposition (e.g. relative to image sensors 110) based on one or more of:sensor/actuator information 167, configuration information 177,instrument state 165, and/or user motion input 143. Configurationinformation 177 may provide information pertaining to the instruments105 coupled robotic medical system 100, image sensor configuration (e.g.lens focal length and other parameters), user preferences (e.g.sensitivity to user movement, the desired level of haptic feedback 141,display parameters, disallowed states, etc.) and/or an operationalconfiguration or mode of operation of robotic medical system 100.

In some embodiments, motion models block 175 may also usesensor/actuator information 167 from one or more sensors and actuatorsin robotic medical system 100. The sensors may include one or more of:electronic sensors; electromagnetic sensors; electro-mechanical sensors,including micro-electro mechanical sensors (MEMS). The sensors may beused to sense actuator articulation/motion of the main sheath, and/orimage sensors 110 and/or the image sensor sub-arm, and/or instruments105 and/or the instrument sub-arms. The sensors may include 3D shapesensing fiber optic sensors; fiber optic force and/or pressure sensorssuch as photonic crystal fiber (PCF) sensors or Fiber Bragg Grating(FBG) sensors, or make use of scattering arising from FBG sensors,inherently present, or make use of post-process produced Rayleighscattering. In some embodiments, sensor/Actuator information 167 fromone or more sensors may be used in conjunction with captured images todetermine instrument pose and/or relative pose.

In some embodiments, the sensors may facilitate instrument pose and/orrobot arm and/or staging kinematic chain S 118 configurationdetermination. Electromagnetic sensors may be embedded in instruments105 and/or at one or more locations/joints in robotic arms 112 (such aselectromagnetic sensor 122), staging kinematic chain S 118, and at otherlocations. Electromagnetic sensors may use an electromagnetic fieldgenerator and small electromagnetic coils to track the instruments.Input from the electromagnetic sensors may be processed (e.g. usingsignal processing techniques) to determine and track the poses of one ormore instruments 105. In some embodiments, signal processing techniquesmay compensate for distortions in sensor readings that may be caused bythe presence of non-magnetic conductive materials in the environment.Electromagnetic tracking and pose determination techniques may operateto determine instrument pose and/or robot arm configuration and/orstaging kinematic chain configuration even in situations where there isno line of sight to instruments 105. In some embodiments, input from theelectromagnetic sensors may be used by control model 175 to determine apose (or relative pose) of one or more instruments 105.

Although shown as separate from processor(s) 150, memory 170 may beexternal and/or internal to processor(s) 150 and may include primaryand/or secondary memory. Program code may be stored in memory 170, andread and executed by processor(s) 150 to perform the techniquesdisclosed herein. As used herein, the term “memory” refers to any typeof long term, short term, volatile, nonvolatile, or other memory and isnot to be limited to any particular type of memory or number ofmemories, or type of media upon which memory is stored. Examples ofstorage media include computer-readable media encoded with databases,data structures, etc. and computer-readable media encoded with computerprograms. Computer-readable media may include physical computer storagemedia. A storage medium may be any available medium that can be accessedby a computer. By way of example, and not limitation, suchcomputer-readable media can comprise Random Access Memory (RAM) andvariations thereof including Non-Volatile RAM (NVRAM), Read Only Memory(ROM) and variations thereof Erasable Programmable (EPROM), FlashMemory, etc. Computer-readable media may also include Compact Disc ROM(CD-ROM), memory cards, portable drives, or other optical disk storage,magnetic disk storage, solid state drives, other storage devices, or anyother medium that can be used to store desired program code in the formof instructions and/or data structures and that can be accessed by acomputer; disk and disc, as used herein, includes compact disc (CD),laser disc, optical disc, digital versatile disc (DVD), floppy disk andBlu-ray disc where disks usually reproduce data magnetically, whilediscs reproduce data optically with lasers. Combinations of the aboveshould also be included within the scope of computer-readable media.

In some embodiments, processor(s) 150 may process robotic control input147 and user motion input 143. The processed information may be sent torobot control system 160 (e.g. via wireless communications interface 102a and/or wired communications interface 102 b). Robot control system 160may process the information received from processor(s) 150 and sendsignals to appropriate actuators on robotic arms 112 and instruments 105coupled to the robot arms to control, articulate/move, retract, deploy,and/or invoke functionality associated with one or more of: robot arms112, image sensors 104, and/or instruments 105.

Although shown in FIG. 1D as discrete blocks, processor(s) 150 and/ormemory 170 may be distributed between platform 110 (e.g. between robotcontrol system 160) and user console 125 (e.g. which may include visualinterface 135), and user interface 140. In one embodiment, platform 110,user interface 140, and console 125 may each have individual localprocessors 150 and/or local memory. For example, when console 125 isremotely situated from platform 110 and/or user interface 140, console125 and platform may each have local processors 150. Accordingly, in oneembodiment, local processors associated with user interface 140 may beconfigured to: (a) obtain and transmit user motion input 143 and robotcontrol input 147 to local processors associated with platform 110 (e.g.robot control system 160) and console 125; and (b) receive hapticfeedback 141 via haptic interface 142 (e.g. based on input received fromrobot control system 160 based on one or more of: instrument state 165,sensor/actuator information 167, captured images 115). Further, console125 may receive and display captured images 115 and provide visualfeedback 117 (e.g. to reflect user motion input) using visual interface135. Conversely, local processors associated with platform 110/robotcontrol system 160 may be configured to: (a) receive user motion input143 from local processors associated with user interface 140, and (b)robot control input 147 from local processors associated with console125. Local processors associated with platform 110/robot control system160 may be configured to provide appropriate motion control/instrumentcontrol input 162 to robot arms 112, instruments 105 and othercomponents coupled to platform 110 (e.g. based on the received usermotion input 143 and robotic medical device control input 147); and (b)obtain and transmit captured images 115, instrument state 165, andsensor/actuator information 167 (and/or determined image sensor pose,instrument pose information, etc.) to local processors associated withuser console 130 and haptic feedback 141 to local processors associatedwith user interface 140. As another example, memory 170 and somefunctionality associated with processor(s) 150 may be shared betweenconsole 125, platform 110, and user interface 140.

In some embodiments, robot control system 160 may obtain sensor/actuatorinformation 167 from sensors/actuators on robotic medical device 200,captured images 115 from image sensors 110, and instrument state 165.Sensor/actuator information 167, captured images 115, and instrumentstate 165 may also be received by processor(s) 150 either directly (whencoupled to platform 110) or indirectly from robot control system 160(e.g. over wireless communication interface 102 a/wired communicationsinterface 102 b). Robot control system 160 may control actuators and/orother electronic, electrical, and electro-mechanical componentsassociated with robot arms 112, staging kinematic chain S 118, and/orinstruments 105 based on the received commands (e.g. user motion input143 and robot control input 147). In some embodiments, robot controlsystem 160 may include functionality to detect and prevent collisions,entanglements, and/or physical contract between robot arms 112 and/orinstruments 105. In some embodiments, robot control system 160 mayautonomously configure actuators, and/or robot arms 112 and/or stagingkinematic chain S 118 based on user motion input 143. For example, inresponse to user motion input 143 reflecting desired poses P ofinstruments E 105 coupled to robotic arms A 112, motion models 175,configuration information 177, and motion logic 179 may be used todetermine configurations C_(w) of robot arms A 112, and/or aconfiguration C_(S) of staging kinematic chain S 118 (e.g. in jointspace) to effectuate the desired poses. In some embodiments,configuration information 177 may include system metrics, actuatorparameter ranges, instrument state 165 and/or sensor/actuatorinformation 167 (e.g. actuator values), which may be used to informdetermination of the configurations C_(w) and selection of a finalconfiguration (e.g. C₁ or C₂).

The methodologies described herein may be implemented in hardware,firmware, software, or any combination thereof. For a hardwareimplementation, the processor(s) 150 may be implemented within one ormore application specific integrated circuits (ASICs), digital signalprocessors (DSPs), image processors, digital signal processing devices(DSPDs), programmable logic devices (PLDs), field programmable gatearrays (FPGAs), processors, controllers, micro-controllers,microprocessors, electronic devices, other electronic units designed toperform the functions described herein, or any combination thereof. Insome embodiments, processor(s) 150 may include capabilities to: processkinematic models, determine poses and/or relative poses of instruments105, image sensor 104, etc., process images, determine and trackcorresponding positions of instruments relative to a current imagesensor pose, provide input to control haptic feedback 141, providevisual feedback 117, and provide input to control actuators on robotarms 112 and/or staging kinematic chain S 118 and/or instruments 105 toeffectuate corresponding desired instrument poses. Processor(s) 150 mayalso include functionality perform other well-known computer vision andimage processing functions such as feature extraction from images, imagecomparison, image matching, object recognition and tracking, imagecompression and decompression, etc.

In some embodiments, the communicative coupling between console 125platform 110 and/or user interface 140 may be wired (e.g. over wiredcommunications interface 102 b) or wireless (e.g. over wirelesscommunication interface 102 a). For example, commands input by userusing console 125 may be wirelessly communicated (e.g. over wirelesscommunication link 102 a) to an robot control system 160, which may formpart of platform 110. In some embodiments, the robot control system maycontrol and drive robot arms 112, staging kinematic chain S 118, andinstruments 105 based on commands received over the communicationsinterface (wired and/or wireless). Wireless communication may includecommunication over Wireless Local Area Networks (WLAN), which may bebased on the IEEE 802.11 standards, and/or over Wireless Wide AreaNetworks (WWAN), which may be based on cellular communication standardssuch as a Fifth Generation (5G) network, or Long Term Evolution (LTE).5G and LTE are described in documents available from an organizationknown as the 3rd Generation Partnership Project (3GPP).

Thus, in some embodiments, console 125 and user interface 140 may beremotely situated from platform 110, and platform 110 may be controlledand operated based on input received by robot control system 160 overthe communications interface. As outlined above, robot control system160 may control actuators and/or other electronic, electrical, andelectro-mechanical components based on the received commands (e.g. fromuser interface 140/console 125 over communications interface).

In situations where there is an increased danger of risk to medicalpractitioners or patients (e.g. from infectious/contagious diseasesetc.) user interface 140 communicatively coupled (e.g. via wiredcommunications interface 102 b and/or wireless communication interface102 a) to robotic medical system 100 may be used at some specifieddistance or through barriers (e.g. such as optically transparent butbiologically protective barriers) to perform medical procedures on thepatient while maintaining safety and distancing protocols.

Further, in situations when skilled practitioners are unavailable (e.g.in remote locations), user interface 140 communicatively coupled (e.g.via wired communications interface 102 b and/or wireless communicationinterface 102 a) to robotic medical system 100 may be usedtelesurgically (e.g. via wired communications interface 102 b orwireless communication interfaces 102 a) to perform or guide performanceof medical procedures (e.g. using locally available resources). Forexample, local medical practitioner(s) may monitor and supervise patientcondition and/or the procedure being performed in accordance with anylocal regulations and/or safety protocols. In the example above, amedical facility deploying robotic medical system 100 may be able to usea remote first medical practitioner (e.g. at a remote first medicalfacility) for one medical procedure using a first user interface 140-1(e.g. telesurgically), and use a second medical practitioner (e.g. at aremote second medical facility) for another medical procedure using asecond user interface 140-2 telesurgically.

FIGS. 2A and 2B show views illustrating the deployment of robotic armson example robotic medical system 100 during an example medicalprocedure. As shown in FIG. 2A, operator (e.g. a physician) 220 may viewcaptured images 215 (e.g. captured by image sensors 104 coupled to anendoscope 108-1) on visual interface 135 coupled to console 125. Basedon the displayed image 215, operator 220 may provide user motion input143 (not shown in FIG. 2A) using user interface 140 to effectuate adesired pose of an instrument 105-1 coupled to endoscope 108-1, whichhas been coupled distal end of 116-1 of robotic arm A₁ 112-1. As shownin FIG. 2A, endoscope 108-1 may be a bronchoscope, which has beeninserted through the nasal cavity into patient 210 using adapter 124.

FIG. 2B shows another view illustrating the deployment of robotic armson example robotic medical system 100 during the example medicalprocedure of FIG. 2A. As shown in FIG. 2B, operator (e.g. a physician)220 may view captured images 215 (e.g. captured by image sensors 104coupled to an endoscope 108-1) on visual interface 135 coupled toconsole 125. Based on the displayed image 215, operator 220 may provideuser motion input 143 (not shown in FIG. 2A) using user interface 140 toeffectuate a desired pose of an instrument 105-1 coupled to endoscope108-1, which has been coupled distal end of 116-1 of robotic arm A₁112-1 on platform 110. FIG. 2B shows staging kinematic chain S 118coupled to platform 110. As shown in FIG. 2B, staging kinematic chain S118 may be capable of movement along axis 120 during the medicalprocedure. In general, the number of degrees of freedom available tostaging kinematic chain may vary and depend on the type of coupling.

Autonomous control of robot arms 112, staging kinematic chain S 118, andinstruments 105 by processor(s) 150 on platform 105 in response to usermotion input 143 allows the operator to focus on the medical procedurebeing performed and removes the cognitive load associated with robot armand/or staging kinematic chain control. Moreover, in embodiments whereplatform 110, console 125, and user interface 140 are housed in distinctunits that are communicatively coupled, physicians/operators may be ableto perform, direct, or guide medical procedures remotely. For example,user commands (via user interface 140) may be relayed over acommunications network (e.g. wireless—such as a 5G network) to controlinstruments poses and functions, and images captured by cameras 104 andother information may be relayed via the communications network toconsole 125 and displayed via visual interface 135.

FIG. 3A shows a flowchart of a method 300 to facilitate medical robotmotion control during medical procedures. In some embodiments, method300 may be based on dynamic online kinematic adaptation of arms and/orinstruments coupled to a robotic medical system (e.g. robotic medicalsystem 100). In some embodiments, method 300 may be performed by roboticmedical system 100 and/or processor(s) 150 and/or robot control system160 in robotic medical system 100. The term “online” refers to thecapability to perform method 300 during a medical procedure. The term“dynamic” refers to the capability to change from one operation mode(e.g. based on a first motion model) to another (e.g. to a second motionmodel) during the course of effectuating desired poses of one or moreinstruments coupled to robotic medical system 100. As outlined above, insome embodiments, the operation mode change may occur within a controltick. In some embodiments, the method 300 may be performed automaticallyin a user-transparent manner based on the corresponding desired poses ofinstruments coupled to the robotic arms and without further user-input.

In some embodiments, method 300 may be performed on robotic medicalsystem (e.g. robotic medical system 100) comprising a staging kinematicchain (e.g. staging kinematic chain S 118) capable of motion with one ormore degrees of freedom (DOF), wherein the staging kinematic chain S 118is coupled to a plurality of independently articulable robotic armsA={A_(r)|1≤r≤N, N≥2}, where A_(r) 112 is a robotic arm and N representsthe number of robotic arms.

In block 310, (e.g. at a first time) a first configuration of therobotic arms C₁={C_(r_1)|1≤r≤N} to effectuate corresponding desiredposes of instruments P_(k_r) coupled to the robotic arms A 112 may bedetermined based on a first inverse kinematic model that includes therobotic arms and assumes that the staging kinematic chain is static. Insome embodiments, each robotic arm A_(r) 112-r may be coupled to adistinct corresponding instrument E_(r) 105-r and each instrument E_(r)105-r may be associated with a corresponding desired pose P_(r). Asoutline above, the first configuration of the independent arms C₁ may bedetermined based on a first inverse kinematic model that includes therobotic arms A 112 and assumes a static staging kinematic chain S 118.For example, in block 310, configuration C₁, to effect the desired posesP of instruments E 105 coupled to robot arms A 112, may be determinedwithout movement of staging kinematic chain S 118. In some embodiments,input from motion models 175 and configuration information 177 (e.g.current states of actuators and/or current robotic arm configuration,current staging kinematic chain configuration, current staging kinematicchain pose, etc.) may be used to determine the first correspondingconfiguration C₁.

In some embodiments, (e.g. at time t1) the first inverse kinematic modelmay determine the first corresponding configuration C₁ of the roboticarms A 112 by determining, for each robotic arm A_(r) 112-r,corresponding robotic arm/joint configurationsC_(1_r)={c_(1r_f)|1≤f≤F_(r)}, where c_(1r_f) represents the firstconfiguration of a joint f associated with robotic arm A_(r) 112-r andF_(r) represents the number of joints on robotic arm A_(r). Each roboticarm A_(r) 112-r may comprise one or more joints so that 1≤f≤F_(r). Therobotic arm configuration may correspond to actuator states/positionsfor actuators associated with each joint f on robotic arm A_(r) 112-r.Accordingly, the corresponding actuator positions for each robotic armA_(r) 112-r may be determined independently of any other robotic armA_(u) 112-u, u≠r and independently of the staging kinematic chain S 118.Thus, the first inverse kinematic model may be based on (N) independentkinematic chains, wherein a kinematic chain r corresponds to a roboticarm A_(r) 112-r in the plurality of robotic arms A.

Accordingly, in some embodiments, the first inverse kinematic model maydetermine the first configuration C₁ of the robotic arms by determining,for each of the robotic arms, corresponding actuator positions. For eachof the robotic arms, the corresponding actuator positions may bedetermined independently of the other robotic arms and independently ofthe staging kinematic chain. Thus, the first inverse kinematic model maybe determined based on a plurality of independent kinematic chains,wherein each independent kinematic chain corresponds to a distinctrobotic arm and the number of independent kinematic chains may be equalto the total number of the robotic arms.

A Jacobian matrix J_(r), may be used to describe the linearization ofthe functional mapping between the position of an instrument (endeffector) coupled to a robotic arm A_(r) 112-r and the correspondingrobot arm joint position.

Mathematically, the first inverse kinematic model for robotic arm A_(r)112-r may be described based on the Jacobian matrix J_(r) where

$\begin{matrix}{{J_{r} = \frac{\partial x_{r}}{\partial C_{r}}},} & (1) \\{{\overset{.}{C}}_{r}^{d} = {(J)^{- 1}{\overset{.}{x}}_{r}^{d}}} & (2)\end{matrix}$

where,x_(r) is the actual (current) Cartesian pose (X_(r), Y_(r), Z_(r)) of aninstrument coupled to robotic arm A_(r) 112-r,C_(r) is the actual (current) robot arm configuration (jointconfiguration) of A_(r) 112-r in joint space,Ċ_(r) ^(d) is the desired joint velocity for A_(r) 112-r and isdetermined for each robotic arm A_(r) 112-r independently, and{dot over (x)}_(r) ^(d) is the desired Cartesian instrument velocity forA_(r) 112-r. Similar equations may be used to determine the orientationsassociated with each robotic arm A_(r) 112-r independently.

In block 320, a set of control parameter values associated with eachrobotic arm based on the first corresponding configuration may bedetermined.

Each robotic arm A_(r) in configuration C₁ may be associated with a setof control parameter values V_(r1)={ν_(r1q)|1≤q≤Q_(r), Q_(r)≥1}, so thatfor robotic medical system 100, the set of control parameter values forthe first configuration C₁ (e.g. based on the first inverse kinematicmodel), of the robotic arms A 112 may be written as V₁={V_(r1)|1≤r≤N}.

In some embodiments, the set of control parameters may comprise one ormore of: (a) for each robotic arm, corresponding available degrees offreedom (DoFs); or (b) for each robotic arm, corresponding ranges ofmotion available to one or more actuators based on a currentconfiguration of the robotic arm; or (c) medical procedure constraintsthat limit the motion of one or more of the robotic arms; or (d) foreach robotic arm, one or more metrics characterizing a correspondingsingularity of the robotic arm or corresponding derivatives of thesingularity of the robotic arm; or (e) one or more metricscharacterizing a singularity of the robotic medical system, orderivatives of the singularity of the robotic medical system includingthe plurality of robotic arms and the staging kinematic chain; or (f)for each robotic arm, one or more corresponding metrics characterizingthe dexterity of the robotic arm, or corresponding derivatives of thedexterity of the robotic arm; or (g) one or more metrics characterizingthe dexterity of the robotic medical system, or derivatives of thedexterity of the of the robotic medical system including the pluralityof robotic arms and the staging kinematic chain. The control parametersabove are merely examples and various other parameters could be used todetermine when to switch from the first inverse kinematic model to thesecond inverse kinematic model.

In some embodiments, the determination of metrics for singularity and/ordexterity may be based on computing the singular values of the Jacobianmatrix. For example, a first method may use a conditional number, whichmay be a ratio

$\left( \frac{\Sigma_{\max}}{\Sigma_{\min}} \right)$

of the largest singular value (Σ_(max)) to the smallest singular value(Σ_(min)). A second method may use the manipulability μ, which refers tothe product of the singular values or equivalents may be computed as√{square root over (|JJ^(T)|)}. Derivatives of the metrics may be thepartial derivatives of the above metrics such as

$\frac{\partial\mu}{\partial C}.$

In some embodiments, in block 330, in response to the value of at leastone determined control parameter ν_(rq) falling outside a correspondingcontrol parameter range ((e.g. ν_(rq)<T1 or ν_(rq)>T2), a pose of thestaging kinematic chain (P_(S)) and a second corresponding configurationof the robotic arms C₂={C_(2_r), |1≤r≤N} and a configuration C_(S) ofstaging kinematic chain S 118 to effectuate the corresponding desiredposes P_(k_Ar) may be determined, wherein the second correspondingconfiguration of the robotic arms C₂ and configuration C_(S) of stagingkinematic chain S 118 is determined based on a second inverse kinematicmodel that includes the robotic arms and assumes a mobile stagingkinematic chain. Configuration Cs may correspond to a pose Ps of thestaging kinematic chain S.

Mathematically, the second inverse kinematic model for robotic medicalsystem 100 may be described in terms of the Jacobian matrix J_(C), where

$\begin{matrix}{{J_{A} = \begin{bmatrix}J_{l} & 0 & \text{...} & 0 & J_{S} \\0 & J_{2} & \text{...} & 0 & J_{S} \\0 & 0 & \ddots & 0 & J_{S} \\0 & 0 & \text{...} & J_{N} & J_{S}\end{bmatrix}},} & (3)\end{matrix}$

where,J_(r), 1≤r≤N, is the Jacobian matrix for robotic arm A_(r) 112-rJ_(S) is the Jacobian matrix for staging kinematic chain S 118,Further, if x_(A) ^(d) represents the combined desired Cartesian poses Pfor all instruments coupled to robotic arms A 112 and {dot over (x)}_(A)^(d) represents the combined desired Cartesian velocities for allinstruments coupled to robotic arms A 112, then

$\begin{matrix}{{x_{A}^{d} = \begin{bmatrix}x_{1}^{d} \\\vdots \\x_{N}^{d}\end{bmatrix}},{{\overset{.}{x}}_{A}^{d} = \begin{bmatrix}{\overset{.}{x}}_{1}^{d} \\\vdots \\{\overset{.}{x}}_{N}^{d}\end{bmatrix}},{C_{A}^{d} = \begin{bmatrix}c_{1}^{d} \\\vdots \\c_{N}^{d} \\c_{S}^{d}\end{bmatrix}},{{{and}\mspace{14mu}{\overset{.}{C}}_{A}^{d}} = \begin{bmatrix}{\overset{.}{c}}_{1}^{d} \\\vdots \\{\overset{.}{c}}_{N}^{d} \\{\overset{.}{c}}_{S}^{d}\end{bmatrix}},{{{and}\mspace{20mu}{\overset{.}{C}}_{A}^{d}} = {\left( J_{A} \right)^{- 1}{\overset{.}{x}}_{A}^{d}}}} & (4)\end{matrix}$

The equations above may be used, in the second inverse kinematic model,to determine the desired configuration of all arms A 112 and of thestaging kinematic chain S 118. Similar equations may be used todetermine the orientations associated with all arms A 112 and theorientations of the staging kinematic chain S 118.

Thus, in step 330, in response to at least one determined controlparameter value falling outside a corresponding control parameter range,a pose of the staging kinematic chain and a second configuration of therobotic arms to effectuate the one or more corresponding desired posesof the one or more instruments may be determined, wherein the secondconfiguration may be determined based on a second inverse kinematicmodel that includes the robotic arms and assumes that the stagingkinematic chain is mobile.

Accordingly, the second inverse kinematic model determines the secondcorresponding configuration of the robotic arms based on a singlekinematic chain that combines available degrees of freedom (DoFs)corresponding to all the sub-arms and available DoFs of the stagingkinematic chain.

FIGS. 3B and 3C shows platform 110 with staging kinematic chain S 118being moved from an initial position (in FIG. 3B) to a subsequentposition (in FIG. 3C) dynamically during a medical procedure inaccordance with certain embodiments disclosed herein.

FIG. 3B shows platform 110 with staging kinematic chain S 118 inposition PS-1 350. FIG. 3C shows platform 110 with staging kinematicchain S 118 in subsequent position PS-2 355. As shown in FIG. 3C, inresponse to a determination that one or more control parameters may falloutside a pre-determined range when robotic arms A 112 are configured asper the first configuration, a second configuration may be determinedand, staging kinematic chain S 118 may be moved to position PS-2 355(from prior position PS-1 350) in accordance with the secondconfiguration to effect the desired poses of instruments coupled torobotic arms A 112. The movement of staging kinematic chain S 118 may bemoved and repositioned when: (a) robotic medical system is online and(b) dynamically, in response to a determination that one or more controlparameters may fall outside a pre-determined range when robotic arms A112 are configured as per the first configuration. For example, theswitch from the first corresponding configuration of the robotic arms(e.g. as determined in block 310) and the second correspondingconfiguration of the robotic arms (e.g. as determined in block 330) maybe effectuated dynamically during a medical procedure being performed.In some embodiments, in method 300, the corresponding desired poses ofinstruments coupled to the robotic arms may be effectuated within acontrol tick.

In some embodiments, method 300 may further comprise (e.g. followingstep 330) determining, based on policies associated with a medicalprocedure being performed, whether the corresponding desired poses ofthe instruments coupled to the robotic arms can be effectuated based onthe pose of the staging kinematic chain and the second correspondingconfiguration of the robotic arms; and in response to a determinationthat the pose of the staging kinematic chain and the secondcorresponding configuration of the robotic arms would result in at leastone policy violation, effectuating the corresponding desired poses ofthe instruments in accordance with the first corresponding configurationof the robotic arms without movement of the staging kinematic chain.

In some embodiments, method 300 may further comprise dynamicallyswitching, upon effectuation of the corresponding desired poses ofinstruments coupled to the robotic arms, to the first inverse kinematicmodel. For example, the first inverse kinematic model may be a defaultoption, and the second inverse kinematic model may be invoked wheneverthe value of at least one determined control parameter ν_(rq) fallsoutside a corresponding control parameter range.

Thus, in some embodiments, method 300 may dynamically revert or switchback to the first inverse kinematic model after effectuating thecorresponding desired poses of the instruments based on the secondinverse kinematic model. In some embodiments, the method may beperformed automatically during a medical procedure based on the one ormore corresponding desired poses of the one or more instruments andwithout further user-input. For example, the switching between the firstinverse kinematic model and the second inverse kinematic model may occurin a manner transparent to the user. User motion input 143 mayautomatically trigger a switch from the first inverse kinematic model tothe second inverse kinematic model (or vice versa) without anyadditional user input. Thus, the staging kinematic chain 118 may beautomatically moved (e.g. based on the second inverse kinematic model)as appropriate based on user motion input thereby facilitating operatorfocus on the medical procedure and relieving operators of the cognitiveload associated with system configuration and motion.

In some embodiments, method 300 may be performed on robotic medicalsystem 100, which may comprise: a staging kinematic chain S 118, capableof motion with one or more degrees of freedom (DOF), a plurality ofinstruments 105, a plurality of independently articulable robotic arms A112, wherein a proximal end of each robotic arm 112-r is coupled to thestaging kinematic chain S 118, wherein a distal end of each robotic arm112-r is coupled to at least one corresponding instrument 105, and aprocessor 150 operationally coupled to the staging kinematic chain S118, the plurality of robotic arms A 112, and the plurality ofinstruments 105, wherein the processor is configured to perform method300 as outlined above.

FIG. 4 shows an exemplary computing subsystem 400 to facilitate medicalrobot motion control during medical procedures. Computing subsystem 400may form a part of robotic medical system 100. For example, computingsubsystem 400 may form part of platform 110, and/or robot control system160, and/or console 125 may be operationally coupled to robot arms 112,and/or staging kinematic chain 118, and/or instruments 105 and/or visualinterface 134 and/or user interface 140.

As shown in FIG. 4, computing subsystem 400 may include processor(s)150, memory 170, and communications interface 402, which may beconnected using connections 406. Connections 406 may take the form ofbuses, lines, fibers, electronic interfaces, links, etc., which mayoperationally couple the above components.

Communications interface 402 may be capable of wired (e.g. using wiredcommunications interface 102 b) or wireless (e.g. using wirelesscommunication interface 102 a) communications with another device orcomponent (e.g., user interface 140, and/or console 125). Capturedimages 115, instrument state 165 (which may include poses ofinstruments), robot arm configuration information, sensor/actuatorinformation 167, staging kinematic chain pose, etc., may be receivedover communications interface 402. User input may also be transmitted(e.g. when computing subsystem is part of user interface 140 and/orconsole 125) or received (e.g. when computing subsystem forms part ofrobot control system 160) using communications interface 402. Wirelesscommunication may include communication over Wireless Local AreaNetworks (WLAN), which may be based on the IEEE 802.11 standards, and/orover Wireless Wide Area Networks (WWAN), which may be based on cellularcommunication standards such as a Fifth Generation (5G) network, or LongTerm Evolution (LTE).

Computing subsystem 400 may also include control interface 408, whichmay provide control input (e.g. to activate, select, deploy, deactivate,move, orient, retract, extend, etc.) and command input (e.g. to exercisefunctions) that drives robotic arms 112, staging kinematic chains,and/or instruments 105. In some embodiments, control interface 408 mayalso output haptic feedback 141 (e.g. to indicate instrument stateand/or guide user input related to instrument movement direction).Control interface 408 may communicate with processor(s) 150 and may becontrolled by processor(s) 150.

Computing subsystem 400 may also include display interface 410, whichmay interact with display 135 (e.g. a 3D or stereoscopic display) toprovide visual feedback 117 (e.g. configuration information, instrumentposition related information, procedure related information, systemstate information, etc.). For example, display interface may generategraphics, and/or other visualization, which may augment or overlay thecaptured images 115. Display interface 410 may communicate withprocessor(s) 150 and may be controlled by processor(s) 150.

In some embodiments, memory 170 may comprise main or primary memory(e.g. RAM) and storage 460. Program code may be stored in memory 170,and read and executed by processor(s) 150 to perform the techniquesdisclosed herein. Storage 460 may include ROM, EPROM, NVRAM, flashmemory, secondary storage, and other computer readable media (e.g. fixedand/or removable drives, optical disks, etc.). Computer-readable media420 may be encoded with databases, data structures, etc. and/or withcomputer programs. By way of example, and not limitation, suchcomputer-readable media may also include CD-ROM, memory cards, portabledrives, or other optical disk storage, magnetic disk storage, solidstate drives, other storage devices, or any other medium that can beused to store desired program code in the form of instructions and/ordata structures and that can be accessed by a computer.

In some embodiments, images captured by image sensors 104, instrumentstate 165, sensor/actuator information 167, motion control/instrumentcontrol input 162, configuration information 177, instrument poses,robotic arm configuration, staging kinematic chain pose, proceduralpolicies that affect robotic operation, control parameter ranges, andother information pertaining to robotic medical system 100 may be storedin memory 170 for operational, training, logging, and other purposes.For example, based on user input and robotic medical systemconfiguration, a procedure performed by robotic medical system 100 maybe recorded and replayed/analyzed at a subsequent time. As anotherexample, the procedure may be live streamed via communications interface402 (e.g. for educational or training purposes).

Memory 170 may motion models 175, which may include kinematic andinverse kinematic models for robotic medical system 100. In someembodiments, motion models 175 may also include calibrated instrumentcontrol models, which may be used to estimate an instrument pose basedon one or more of: motion control/instrument control input 162,configuration information 177, and/or user motion input 143. In someembodiments, control model 175 may also use information from one or moresensors (when present) in robotic medical system 100. Motion models mayalso include information to determine robotic arm configuration andstaging kinematic chain pose and staging kinematic chain configuration.

Memory 170 may include configuration information 177, which may provideinformation pertaining to the instruments on robotic medical system 100,image sensor configuration (e.g. lens focal length and otherparameters), user preferences (e.g. sensitivity to user movement, thedesired level of haptic feedback 141, display parameters, etc.) and/oran operational configuration or mode of operation of robotic medicalsystem 100 (e.g. indicating whether staging kinematic chain S 118 may bemoved during a procedure or a portion of the procedure).

The methodologies described herein may be implemented in hardware,firmware, software, or any combination thereof. For a hardwareimplementation, the processor(s) 150 may be implemented within one ormore application specific integrated circuits (ASICs), digital signalprocessors (DSPs), image processors, digital signal processing devices(DSPDs), programmable logic devices (PLDs), field programmable gatearrays (FPGAs), processors, controllers, micro-controllers,microprocessors, electronic devices, other electronic units designed toperform the functions described herein, or any combination thereof. Insome embodiments, processor(s) 150 may include capabilities to: processmotion models including kinematic models, determine robot armconfiguration, staging kinematic chain configuration, determine actuatorinput to effect a determined arm configuration, determine robot armconfigurations to effect an instrument pose, determine an instrumentpose from robot arm configuration, etc. Processor(s) 150 may alsoinclude functionality perform other well-known computer vision and imageprocessing functions such as feature extraction from images, imagecomparison, image matching, object recognition and tracking, imagecompression and decompression, etc.

Although the present disclosure is described in connection with specificembodiments for instructional purposes, the disclosure is not limitedthereto. Various adaptations and modifications may be made to thedisclosure without departing from the scope. Therefore, the spirit andscope of the appended claims should not be limited to the foregoingdescription.

What is claimed is:
 1. A method on a robotic medical system comprising astaging kinematic chain capable of motion with one or more degrees offreedom (DOF), wherein the staging kinematic chain is coupled to aplurality of independently articulable robotic arms, the methodcomprising: determining a first configuration of the robotic arms basedon a first inverse kinematic model that includes the robotic arms andassumes that the staging kinematic chain is static, wherein the firstconfiguration of the robotic arms effectuates one or more correspondingdesired poses of one or more instruments coupled to the robotic arms;determining a set of control parameter values associated with one ormore of the robotic arms based on the first configuration; anddetermining, in response to at least one determined control parametervalue falling outside a corresponding control parameter range, a pose ofthe staging kinematic chain and a second configuration of the roboticarms to effectuate the one or more corresponding desired poses of theone or more instruments, wherein the second configuration is determinedbased on a second inverse kinematic model that includes the robotic armsand assumes that the staging kinematic chain is mobile.
 2. The method ofclaim 1, wherein the first inverse kinematic model determines the firstconfiguration of the robotic arms by determining, for each of therobotic arms, corresponding actuator positions.
 3. The method of claim2, wherein, for each of the robotic arms, the corresponding actuatorpositions are determined independently of the other robotic arms andindependently of the staging kinematic chain.
 4. The method of claim 1,wherein the first inverse kinematic model is based on a plurality ofindependent kinematic chains, wherein each independent kinematic chaincorresponds to a distinct robotic arm.
 5. The method of claim 4, whereina count of the plurality of independent kinematic chains is equal to atotal number of the robotic arms.
 6. The method of claim 1, furthercomprising: dynamically switching to the first inverse kinematic modelupon effectuation of the one or more corresponding desired poses of theone or more instruments based on the second inverse kinematic model. 7.The method of claim 1, wherein the method is performed automaticallyduring a medical procedure based on the one or more correspondingdesired poses of the one or more instruments and without furtheruser-input.
 8. The method of claim 1, wherein the set of controlparameter values associated with one or more of the robotic armscomprise one or more of: corresponding degrees of freedom (DoFs)available to the one or more robotic arms; or corresponding ranges ofmotion available to one or more actuators based on a currentconfiguration of the one or more robotic arms; or medical procedureconstraints that limit motion of the one or more robotic arms; or one ormore first metrics characterizing a corresponding singularity of atleast one robotic arm or corresponding derivatives of the singularity ofthe at least one robotic arm; or one or more second metricscharacterizing the dexterity of the at least one robotic arm, orcorresponding derivatives of the dexterity of the at least one roboticarm; or one or more third metrics characterizing a singularity of therobotic medical system, or derivatives of the singularity of the roboticmedical system including the plurality of robotic arms and the stagingkinematic chain; or one or more fourth metrics characterizing thedexterity of the robotic medical system, or derivatives of the dexterityof the of the robotic medical system including the plurality of roboticarms and the staging kinematic chain.
 9. The method of claim 1, furthercomprising: determining, based on policies associated with a medicalprocedure being performed, whether the one or more corresponding desiredposes of the one or more instruments can be effectuated based on thepose of the staging kinematic chain and the second configuration of therobotic arms; and in response to a determination that the pose of thestaging kinematic chain and the second configuration of the robotic armswould result in at least one violation of the policies, effectuating thecorresponding desired poses of the one or more instruments in accordancewith the first configuration of the robotic arms without movement of thestaging kinematic chain.
 10. The method of claim 1, wherein secondinverse kinematic model determines the second configuration of therobotic arms based on a single kinematic chain, wherein the singlekinematic model combines available degrees of freedom (DoFs)corresponding to the plurality of robotic arms and available DoFs of thestaging kinematic chain into the single kinematic chain.
 11. The methodof claim 1, further comprising: effectuating either the firstconfiguration of the robotic arms or the second configuration of therobotic arms dynamically during a medical procedure.
 12. The method ofclaim 11, wherein the effectuation of either the first configuration ofthe robotic arms or the second configuration of the robotic arms occurswithin a control tick.
 13. A robotic medical system comprising: astaging kinematic chain, capable of motion with one or more degrees offreedom (DOF), a plurality of independently articulable robotic armscoupled to the staging kinematic chain, one or more instruments coupledto the robotic arms, and a processor operationally coupled to thestaging kinematic chain, the plurality of robotic arms, and the one ormore instruments, wherein the processor is configured to: determine afirst configuration of the robotic arms based on a first inversekinematic model that includes the robotic arms and assumes that thestaging kinematic chain is static, wherein the first configuration ofthe robotic arms effectuates one or more corresponding desired poses ofthe one or more instruments coupled to the robotic arms; determine a setof control parameter values associated with one or more of the roboticarms based on the first configuration; and determine, in response to atleast one determined control parameter value falling outside acorresponding control parameter range, a pose of the staging kinematicchain and a second configuration of the robotic arms to effectuate theone or more corresponding desired poses of the one or more instruments,wherein the second configuration is determined based on a second inversekinematic model that includes the robotic arms and assumes that thestaging kinematic chain is mobile.
 14. The robotic medical system ofclaim 13, wherein the first inverse kinematic model is based on aplurality of independent kinematic chains, wherein each independentkinematic chain corresponds to a distinct robotic arm.
 15. The roboticmedical system of claim 13, wherein the processor is further configuredto: dynamically switch to the first inverse kinematic model uponeffectuation of the one or more corresponding desired poses of the oneor more instruments based on the second inverse kinematic model.
 16. Therobotic medical system of claim 13, wherein the processor is furtherconfigured to: automatically effectuate either the first configurationof the robotic arms or the second configuration of the robotic armsdynamically during a medical procedure.
 17. The method of claim 16,wherein the effectuation of either the first configuration of therobotic arms or the second configuration of the robotic arms occurswithin a control tick.
 18. The robotic medical system of claim 13,wherein the set of control parameter values associated with one or moreof the robotic arms comprise one or more of: corresponding degrees offreedom (DoFs) available to the one or more robotic arms; orcorresponding ranges of motion available to one or more actuators basedon a current configuration of the one or more robotic arms; or medicalprocedure constraints that limit motion of the one or more robotic arms;or one or more first metrics characterizing a corresponding singularityof at least one robotic arm or corresponding derivatives of thesingularity of the at least one robotic arm; or one or more secondmetrics characterizing the dexterity of the at least one robotic arm, orcorresponding derivatives of the dexterity of the at least one roboticarm; or one or more third metrics characterizing a singularity of therobotic medical system, or derivatives of the singularity of the roboticmedical system including the plurality of robotic arms and the stagingkinematic chain; or one or more fourth metrics characterizing thedexterity of the robotic medical system, or derivatives of the dexterityof the of the robotic medical system including the plurality of roboticarms and the staging kinematic chain.
 19. The robotic medical system ofclaim 13, wherein second inverse kinematic model determines the secondconfiguration of the robotic arms based on a single kinematic chain,wherein the single kinematic model combines available degrees of freedom(DoFs) corresponding to the plurality of robotic arms and available DoFsof the staging kinematic chain into the single kinematic chain.
 20. Anon-transitory computer-readable medium comprising instructions toconfigure a processor coupled to a robotic medical system to: determinea first configuration of the robotic arms based on a first inversekinematic model that includes the robotic arms and assumes that thestaging kinematic chain is static, wherein the first configuration ofthe robotic arms effectuates one or more corresponding desired poses ofthe one or more instruments coupled to the robotic arms; determine a setof control parameter values associated with one or more of the roboticarms based on the first configuration; and determine, in response to atleast one determined control parameter value falling outside acorresponding control parameter range, a pose of the staging kinematicchain and a second configuration of the robotic arms to effectuate theone or more corresponding desired poses of the one or more instruments,wherein the second configuration is determined based on a second inversekinematic model that includes the robotic arms and assumes that thestaging kinematic chain is mobile.